dylansmithfandomcom-20200215-history
How to Write High Quality Test Cases
How to Identify Test Cases In general Test Cases are created in 2 scenarios: *Validating new functionality works as intended *Reproducing a defect to validate the resolution Test Cases for new functionality should be authored by the same team that implements the functionality during the same Sprint. The User Stories should drive the creation of the Test Cases; specifically the Acceptance Criteria (or whatever mechanism is used to capture requirement details) should be used to identify necessary Test Cases. As the Stable Team Testers are a primary consumer of the Acceptance Criteria (in addition to Developers) it is expected they be closely involved in the elaboration and review of these Acceptance Criteria (typically performed during Sprint Planning). Test Cases should be identified that prove out the natural path through the functionality, but also Test Cases for the various edge cases that result from the requirement details. For defects that are found as a result of running a Test Case (either during Functional or Release Testing phases), when the defect is created using the MTM tool it will automatically be linked to the Test Case that found it. For defects that are discovered outside of running Test Cases (e.g. reported by a user in production, found during exploratory testing, etc), a Test Case should be created to represent the repro steps and linked to the Defect Work Item. *Include sample User Story + Acceptance Criteria + Test Cases *We need "enough" test cases to build confidence, risky features require more testing Basic Test Case (Title, Steps, Links) The above image is a Test Case in TFS/MTM. The 3 most important pieces of data that belong to a Test Case are pointed out: A - Test Case Title - The title will be used to identify the Test Case when viewing a list of test cases. It should describe the specific scenario that this Test Case is intended to test. It should be descriptive enough to distinguish the Test Case at a glance from other like Test Cases. However, Test Cases will be linked to a User Story, and it can be assumed that the person viewing the Test Case title knows which User Story this Test Case belongs to, so the title only needs to be descriptive enough to differentiate between this Test Case and the other Test Cases belonging to that User Story. *Include example of User Story - Test Case(s), include both good and bad examples *Possibly include screenshot from MTM for the examples B - Test Steps and Expected Results - The Test Steps Actions/Expected Results are the core of the Test Case. These are the step-by-step instructions of the actions to take, and the expected results that should be verified. Not every Action should have an Expected Results, however in the cases where it is used, the expected result should be paired with the Action that causes that result to occur. Every Test Case should have at least one Expected Result (otherwise, what is it actually testing?). The plan is that Stable Team Testers with deep domain knowledge will be the primary test case authors. However, the goal is to have staff with minimal domain knowledge responsible for re-running those Test Cases, and eventually converting them to automated regression tests. For this reason, the Test Steps should be sufficiently detailed that a user with minimal domain/application knowledge can follow the steps and execute the Test Case. *Include screenshot example of good and bad Test Steps C - Test Case Links - Work Item links are the primary mechanism used to organize the library of Test Cases. Test Cases should have a link to the User Story (or Defect) that this Test Case was written to test. That is the typical method that will be used to look up and organize Test Cases. In addition, any Defects that are discovered as a result of executing this Test Case should be linked back to this Test Case. This information is visible in the All Links tab of the Test Case (the Tested User Stories tab contains a subset of this data). These links are used extensively for reporting and analyzing test data (see Reporting for more details on how this is used). *Include screenshot of Test Case with links Test Case Best Practices There are some common qualities that make a test effective: *Independent *Repeatable *Valuable *Small Independent An independent test is one that does not require other tests to be run either before or after - it is stand-alone. Writing independent Test Cases provides flexibility when planning out test runs, as any combination of tests in any order is valid. Repeatable A test should be repeatable and predictable. Meaning that you can run the same test over and over, many times, over many days/weeks, and the results should continue to be the same. This also implies that the Test Case does not modify the testing data in a way that would prevent the test from being run again. For example, if there is a Test Case that deletes a specific Customer, that Test Case must also re-create that Customer at the start or end of the Test Case otherwise it would not be repeatable. Valuable A test should test a scenario that provides value to the team by building confidence that the software is working as intended. This is is often a subjective quality, but as an example there is probably little value in writing a Test Case that has the tester type a value into a field, then simply verify that the value shows up in that field. What would be more valuable is having the Tester enter several values (e.g. creating a new customer), then navigating to a different screen and verifying that all the values input have been saved and are displaying correctly. Ultimately, it should be a decision made by the Stable Team Tester whether a given test case is valuable or not. Small Assuming you have satisfied the Independent, Repeatable and valuable criteria, the next consideration is size of the test. Should we write one big test, or 3 smaller tests? In general, many smaller tests is better, assuming those tests are all independent, repeatable and valuable. Smaller tests give more flexibility in terms of test planning (ability to just plan/run one of the many small tests), and the test results convey more information (now we can have 2 passing tests and one failing test, instead of one giant test that fails. This gives more specific feedback to where the failure is). Often there will be a tradeoff to be made between value and size, this decision should be made by a Stable Team Tester as they have the knowledge necessary to make the best decision. Shared Steps Shared Steps is a feature available in MTM that allows the test author to reuse a set of Test Steps across many Test Cases. It is common for a set of Test Steps to appear in many Test Cases. For example, many tests will start by launching and logging in to the application, which may comprise several Test Steps. Once these common steps have been identified, they can be factored out into something called Shared Steps, that can be reused in many Test Cases. The benefit of using Shared Steps is that when authoring new Test Cases you can easily "drop in" existing Shared Steps instead of writing the steps from scratch (or copy-pasting from other Test Cases). In addition, should the steps ever change (e.g. the flow of the Login screen changes requiring changes to the Test Steps that describe the login process), by using Shared Steps the change only needs to be made in one place - as opposed to if those steps had been copy-pasted between many Test Cases, then each of those Test Cases would need to be updated. There are no concrete rules about when to use or not use Shared Steps. It will be the responsibility of the test author(s) to familiarize themselves with the library of Shared Steps and use them as appropriate. * Include screenshot(s) Test Case Parameters / Iterations What are they When to use When not to use The main advantage of using multiple iterations is the ease of maintenance... The main downside of using multiple iterations is the MTM tooling assumes that when you execute a Test Case you will always execute all iterations. This limits the flexibility for test planning. There is no straightforward way to plan to run only some iterations, it's all or nothing. Test Author(s) will need to make a tradeoff between ease of maintenance and test planning flexibility when deciding whether to combine test scenarios into one Test Case with multiple iterations, or create multiple Test Cases. Should you create many Test Cases, or one Test Case with many iterations? Talks about the tradeoffs between maintainability, planning flexibility, and ease of automation What about Shared Steps and parameters? benefit: puts text in clipboard benefit: makes automation easier as it figures out the bindings Test Case Lifecycle How do test cases get reviewed What is the workflow / states they go through What is the impact if you want to modify a mature test case (automated? reviewed? etc) Do we copy the Test Case when automating? Do we introduce a new link type for "automated by"? Design Ready Inventory Automated (linked to another Test Case) More Test Case Meta Data What are all the other fields on a Test Case